California Department of Education 
Executive Office 
SBE-002 (REV. 11/2017) 


memo-imb-amard-june20item01 


MEMORANDUM 


DATE: June 2, 2020 


TO: MEMBERS, State Board of Education 

FROM: TONY THURMOND, State Superintendent of Public Instruction 

SUBJECT: Update on the Implementation of the Integrated Local, State, and Federal 

Accountability and Continuous Improvement System: Chronicle Review of the 
Work Completed on the Student-Growth Model and a Progress Report on 
Ongoing Work. 


Summary of Key Issues 


At the July 2018 meeting, the State Board of Education (SBE) directed the California 
Department of Education (CDE) to conduct further study on a measure of individual 
student growth, including the impact of future years of assessment data, changes in the 
model to reduce year-to-year volatility, consideration of additional growth models or 
options, and an examination of growth models implemented in other states. 
(https://www.cde.ca.gov/be/ag/ag/yrl8/documents/jull8item01.docx). 

The CDE engaged the California Comprehensive Center (CCC) to conduct this 
research and to facilitate a stakeholder process on the future direction of this work. The 
CCC contracted with the National Center for Improvement of Education Assessment for 
Joseph Martineau to facilitate the discussion with the stakeholder group. Based on the 
feedback gathered from the stakeholder engagement process, the CDE has moved 
forward with a regression-based individual student-level growth model and is engaging 
the Educational Testing Service (ETS) to explore how this model can be best 
implemented. This Information Memorandum chronicles the work (from 2015 to the 
present) to identify an appropriate model to measure student growth; highlights some of 
the decision milestones that have been reached along the way; and charts the tasks 
ahead for the CDE, the ETS, and the Student Growth Stakeholder Group to advance 
this work. 

Attachment(s) 

Attachment 1: Review of the Work Completed on the Student-Growth Model and 

a Progress Report on Ongoing Work (8 pages). 
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Attachment 1 

Summary of Work on Student Growth from 2015-Present 

Initial Work on Student Growth within a New Accountability System 

The initial work on student growth within the accountability context began in 2015 and 
coincided with the development of a new accountability system for the state of California. 

Data from the 2014 scientific sample of the California Assessment of Student Performance 
and Progress (CAASPP) field test and the initial year of 2015 CAASPP results were used to 
conduct various growth model simulations in late 2015. Initial findings were presented to the 
California Department of Education (CDE) Technical Design Group (TDG). 

Discussions of a new state accountability system, based on the Local Control Funding 
Formula (LCFF) state priorities, with the newly enacted federal Every Student Succeeds Act 
(ESSA) accountability requirements, began at the January 2016 State Board of Education 
(SBE) meeting. The CDE informed both the SBE and SBE staff on various options for a 
student-level growth model. In February 2016, the SBE received its first Information 
Memorandum on student-level growth models that could be used to communicate Smarter 
Balanced Summative Assessment results 

( https://www.cde.ca.aov/be/pn/im/documents/memo-dsib-amard-febl6item01.docT 

SBE members requested additional information on student-level growth models at their May 
2016 meeting; specifically, they requested an explanation on how these models would fit into 
the proposed accountability system. CDE responded to these questions in a June 2016 
Information Memorandum, focusing in particular on the differences between student-level 
reporting and school- and district-level accountability reporting 

( https://www.cde.ca.aov/be/pn/im/documents/memo-dsib-amard-junl6item01.docl The 

memorandum addressed some confusion that existed around possible report venues, and 
clarified that student reports generally use simple gain scores that are easy to communicate, 
while accountability systems traditionally use more complex growth models to obtain greater 
validity and reliability. In Fall 2016, when the second year of CAASPP data was released (and 
which marked the first opportunity to examine growth data for the majority of students in the 
state) CDE worked with the TDG and Educational Testing Services (ETS) to examine the 
growth data. 

2017: Exploring Growth Model Options 

In January 2017, the SBE discussed the specific criteria for selecting a growth model for 
school and district accountability 

(https://www.cde.ca.gov/be/ag/ag/yrl7/documents/janl7item02.doc): 

1. Conform to rigorous technical standards. 

The growth model should measure academic progress over time for schools, local 
educational agencies (LEAs), and the state. It should produce precise information that 
is valid for its purpose. The model should have the capacity to produce reliable results 
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for student groups as small as 30. 


2. Be capable of being included in accountability systems. 

The growth model should fit into a multiple measures approach for examining state 
and district academic progress over time, as envisioned in the SBE approved 
accountability system. Additionally, the information should be useful to LEAs in 
establishing local goals and evaluating local programs (e.g., Local Control and 
Accountability Plans). 

3. Provide a measure of academic growth across the continuum of performance. 

The growth model should allow for progress to be measured across the continuum of 
academic achievement. The model should have the capacity to evaluate academic 
achievement gaps between students’ groups in such a way as to make determinations 
about the narrowing of those gaps. The information on which the growth model is 
based should be consistent from year to year and reflect how students performed in 
terms of where they started in the previous year. 

4. Provide for the inclusion of all students. 

The growth model should be applied to all students who earn a valid score on the 
ELA/literacy and mathematics statewide assessments. The information used to 
produce the growth model should be based only on student test scores and not on any 
other school or student characteristics. 

5. Provide information on academic progress that is easily communicated to 
educators and the public. 

Information from the growth model, at the school-, LEA-, and state-level, should be 
displayed in a manner that stakeholder groups can easily understand. 

Following the SBE meeting, the CDE consulted with the ETS, the TDG, the CAASPP 
Technical Advisory Group (TAG), the California Practitioners Advisory Group (CPAG), and the 
Statewide Assessment Stakeholder Group, and various stakeholders regarding potential 
growth models. Table 1 lists the ten growth models that were explored during these meetings 
and the central question that each model attempts to answer. The individual student growth 
models fit into one of three general categories: Absolute Growth, Relative Growth, and 
Projected Growth. 
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Table 1. Growth Models Considered for Simulation Studies 


General Category 

Type of Growth Model 

Central Question 

Absolute Growth* 

1. Gain Score 

How much did a student’s 
performance change from 
last year? 

Absolute Growth* 

2. Average Gain 

How much did a student’s 
performance change (on 
average) over multiple 
years? 

Absolute Growth* 

3. Categorical Model 

How has a student 
progressed towards a 
desired performance level? 

Relative Growth** 

4. Residual Gain Model 

How much higher/ lower did 
a student score this year 
than expected, given past 
performance? 

Relative Growth** 

5. Conditional Percentile 
Ranks of Gains 

How does a student’s gain 
this year compared to that of 
other students who started 
at the same place last year? 

Relative Growth** 

6. Student Growth 
Percentile Model: 
(SGPs) 

How does a student’s 
current performance 
compare to that of other 
students with the same prior 
test scores? 

Projected Growth*** 

7. Trajectory Model 

What is the minimum gain 
score a student must 
maintain to reach a target 
future standard? 

Projected Growth*** 

8. Projection Model 

Where is a student likely to 
score in the future, given the 
student’s score history? 

Projected Growth*** 

9. The SGP Model: 
Projections 

What is the minimum SGP a 
student must maintain to 
reach a target future 
standard? 

Projected Growth*** 

10. Probability Model 

What is the probability that a 
student will reach a desired 
benchmark in a future grade 
level? 

*Background green for abso 

ute growth 


**Background blue for relative growth 
***Background orange for projected growth 
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At the March 2017 SBE Meeting, CDE notified the SBE that it would take twelve to eighteen 
months to complete work on a student-level growth model, and that it would not be ready for 
accountability purposes until the 2018 Dashboard 
(http://www.cde.ca.gov/be/ag/ag/yrl7/documents/marl7item02.doc). 

Based on the criteria set by the SBE at its January 2017 meeting and on the feedback 
received from its stakeholder groups, the CDE narrowed the field of potential growth models 
from ten to three options. It presented these three options to the SBE at the July 2017 
meeting, f https://www.cde.ca.aov/be/pn/im/documents/memo-asb-adad-junl7item03.doc T 

• Change-in-Distance-to-Met (CDTM): Measures absolute growth of each student from 
the prior year to the current year using Distance from Level 3 as the measurement 
threshold. 

• Conditional Percentile Rank of the Gain (CPR): Ranks the growth of students who 
are grouped together as a result of having the same prior year test scores, in the same 
subject and grade. 

• Residual Gain (RG): The difference between a student’s predicted test score and 
actual test score. Note: the predicted score is based on prior ELAand mathematics 
test scores, as well as on the scores of all other students in the same grade. 

Following the July 2017 SBE meeting, CDE directed ETS to complete a statistical analysis of 
all three proposed growth models, using growth data calculated from the 2015 and 2016 
administrations of the CAASPP. The research question(s) for each model included within- and 
across-model investigation, exploring how aggregate scores differed among student groups 
and school composition. 

2018: Finding the Best Fit for Student Growth in California 

Technical work on the student growth analysis was completed by ETS in early 2018, and the 
results of this study were formalized and published in an SBE Information Memorandum in 
February 2018 (https://www.cde.ca.gov/be/pn/im/documents/memo-pptb-amard- 
febl8item01.docx). Additionally, ETS and CDE presented this analysis to the CAASPP 
Technical Advisory Group, the TDG, and CPAG, as well as to LCFF and assessment 
stakeholders for feedback. 

The Student Growth Model was formally introduced at the March 2018 SBE Meeting, as part 
of the CDE’s presentation on the 2018 Dashboard workplan 

(https://www.cde.ca.gov/be/ag/ag/yrl8/documents/marl8item01.docx). At that time, it was 
anticipated that, at its May 2018 meeting, the SBE would select one of three growth models 
for possible inclusion in the 2018 Dashboard, and that it would formally adopt the model at its 
September 2018 meeting. The SBE reviewed the growth models at its May 2018 meeting and 
directed the CDE to conduct additional analyses on the RG model, using data from the spring 
2017 CAASPP administration 

(https://www.cde.ca.gov/be/ag/ag/yrl8/documents/mayl8item02.docx). Specifically, the CDE 
was asked to address two questions: 
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1. What information does the RG model provide compared to the information that the 
current District from Standard (DFS) provides? 

2. How would the RG model fit into the current accountability five-by-five structure? 

The CDE shared its analyses with the SBE in a June 2018 Information Memorandum. The 
memorandum also included the findings from the second stage of the ETS study on growth 
models for California. The memorandum pointed out that the RG model provides information 
on the number of students who met their predicted test scores, as well as the variance 
between their actual and predicted test scores; however, the model does not provide the 
amount of improvement needed to bring the average student up to the Standard Met Level 
(Level 3) on the Smarter Balanced Assessment, whether achievement gaps are closing, or if 
students were on track to reach proficiency. Additionally, the memorandum pointed out that 
replacing change with growth in the current five-by-five structure, which is based on 
continuous improvement, could result in conflicting or confusing messages to the public; for 
example, under the RG model, a school could decrease in Status while experiencing positive 
Growth, and vice-versa. 

The ETS Report, entitled “Updated Analysis of the Residual Gain Model 2017-18 Plan,” was 
based on three years of CAASPP data (2015, 2016, 2017) and two growth measurements 
(2015 to 2016 and 2016 to 2017). The analysis showed that, while the RG model performed 
statistically similar in both years, there was low year-to-year stability within the outcomes. 
“Such high volatility can make it difficult for local educational agencies to use the growth data 
for driving decisions, as decisions made one year might be contradicted with the next year’s 
growth data. It may be more advisable to look for patterns in these scores over several years 
than to act on their values in a given year” 

(https://www.cde.ca.gov/be/pn/im/documents/memo-pptb-amard-junl8item01al.docx).). 

Therefore, at the July 2018 meeting, the SBE directed the CDE to conduct further analyses 
on the RG model and paused the incorporation of any student-level growth model in the 2018 
Dashboard. The SBE wanted to see the impact of future years of assessment data on the 
model, what changes could be made to reduce year-to-year volatility, if there were a 
consideration of additional growth models or options, and how growth models had been 
successfully implemented in other states. Additionally, the SBE requested that the CDE 
convene a stakeholder group to discuss what additional information was wanted from an 
individual student growth model that was not already available in current testing data 
(https://www.cde.ca.gov/be/ag/ag/yrl8/documents/jull8item01.docx). 
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2019: Stakeholder Engagement on the Growth Model 

In response to the SBE’s request, the CDE engaged the California Comprehensive Center 
(CCC) to convene a stakeholders workgroup to advance the work of the growth model 
(https://www.cde.ca.gov/be/pn/im/documents/novl9memoamard01.docx). Nineteen 
participants were gathered from a cross-section of researchers, educators, and staff from 
educational and equity advocacy groups who had been actively contributing to the broader 
discussion of growth models. The CCC organized four in-person meetings to gather a deeper 
understanding of what a growth model could provide California. A summary of the meeting 
dates and topics is shared below in Table 2. 

Table 2: The 2019 Stakeholder Meetings 


Meetings 

Date (2019) 

Attendance 

* 

Purpose 

Meeting #1 

February 4 

14 

Set common understanding of CA 
accountability system, growth model work 
to date, Smarter Balanced scale, and 
terms/language/definitions 

“Catch-up” 

Virtual 

Meeting 

March 11 

5 

On March 11, the CA CC hosted a virtual 
meeting, presenting the same content as 
the February 4 meeting. All five 
stakeholder representatives who did not 
attend the February 4 meeting attended 
this “catch-up” meeting. 

Meeting #2 

April 10 

17 

Explore, analyze, and prioritize potential 
“interpretations” (i.e., data questions that 
can be answered by various growth 
models) 

Meeting #3 

May 29 

14 

Review selected states’ growth models, 
review various analytical models, and 
discuss broad interpretations 

Conversation 
s Kit 

Orientation 

July 8-9 

8 

On July 8 and 9, the CA CC hosted an 
optional virtual meeting to orient 
stakeholder representatives to the tools in 
the Conversations Kit. 

Meeting #4 

September 

23 

15 

Review results from the survey about 
community conversations, discuss most 
important actions and behaviors 


*Attendance counts reflect only stakeholder representatives. Presenters, observers, and staff 
of the CDE, SBE, and CA CC also attended and are not included in the attendance counts. 
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The meetings provided insight into how deeply valued the idea of student growth is within 
California yet how difficult it is to create a growth measure that covers the wide variety of 
information various people hope a growth model can provide. Reflecting on the insights 
gleaned from the collective meetings, the CDE concluded that the RG model best fit the 
expectations and wants expressed by the group. It then set its attention on addressing the 
concerns voiced by the SBE at its July 2018, specifically, the year-to-year volatility of the 
scores. 

2020: Releasing Student Growth Data 

Beginning in February 2020, the CDE reengaged the TDG on this work and specifically, how 
technical modifications could improve the results of the RG model within an accountability 
framework. From this discussion the CDE considered two options: 

• Changing the Highest Obtainable Scale Score (FIOSS)/Lowest Obtainable Scale 
Score (LOSS) on the CAASPP, which would result in allowing more students to show 
gains at the higher thresholds, and 


• Implementing new regression formulas to counter small n-size and stability issues. 

In March 2020, the CDE included in its 2020 Accountability Work Plan, the release of a 
student-level growth model in December 2020. With this timeline in place, CDE began 
meeting with ETS to discuss further analyses of California’s growth data. In April 2020, ETS 
presented the Estimated Best Linear Prediction (EBLP) model to the TDG. Following the 
meeting, ETS and CDE agreed to analyze the latest year of growth data using the RG 
methodology but to also conduct a new analysis of growth using the EBLP methodology. 

ETS will share the results of these analyses with the TDG and the CPAG in August 2020. At 
that time, either the CDE will either accept the results as they are or request (per TDG and 
CPAG guidance), additional model constraints. A summary of the impact of the EBLP 
methodology on stability will be shared at the September 2020 SBE Meeting. 

At the March 2020 meeting, the SBE directed the CDE to provide a presentation for the May 
2020 meeting regarding the work conducted to date on the development of a student-level 
growth model. However, due to the COVID crises and shift in activities for accountability, the 
CDE postponed the presentation until the July 2020 SBE meeting 
(https://www.cde.ca.gov/be/ag/ag/yr20/documents/mar20item05.docx). 
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2020 Timeline for the Development of a Measure of Individual Student Growth 

This table provides a calendar and timeline of the key activities related to the ongoing 
development and implementation of a measure of individual student growth for release in 
December 2020. Note: the timing of these activities may be subject to change. 


Date 

Activity 

February 2020 

Discussion with TDG on technical modifications (HOSS/LOSS 
and EBLP) to RG model 

March 2020 

SBE Meeting: Update on the 2020 Dashboard work plan 

April 2020 

Presentation to the TDG on EBLP modeling and initial technical 
requirements 

April/May 2020 

Discussions with ETS on EBLP and regression modeling 
parameters 

June 2020 

SBE Information Memorandum on the Chronicle Review of the 
work completed on the student growth model, and presentation 
of workplan and EBLP modeling to Student Growth 

Stakeholder Group 

July 10-11, 2020 

SBE Meeting: Item on Measure of Individual Student Growth 

July 2020 

ETS to deliver a technical report to CDE on EBLP and 2018-19 
Regression Modeling 

August 2020 

ETS to present a report to TDG -Discuss any additional 
technical adjustments and the CDE to present the EBLP 
methodology to the CPAG 

August-September 2020 

ETS conducts additional technical work on Growth Model by 
ETS, as requested by TDG 

September 2020 

SBE Meeting: Present preliminary results of the EBLP 
methodology as compared to the prior RG Model and SBE to 
provide CDE with direction on which model to pursue 

October 2020 

Discussion of final ETS report with TDG and provide SBE with 
a report in an Information Memorandum 

November 2020 

SBE Meeting: Approval of Model to Measure Individual Student 
Growth 

December 2020 

Public Release of Growth Model Data (LEA/School/Student 
Group level) on the CDE California School Dashboard and 
System of Support webpage 

















